Papers with Neural Language Models
Do Language Models Make Human-like Predictions about the Coreferents of Italian Anaphoric Zero Pronouns? (2022.coling-1)
Copied to clipboard
| Challenge: | Some languages allow arguments to be omitted in certain contexts, but human language comprehenders construct expectations about which referents are more likely. |
| Approach: | They ask whether Neural Language Models extract expectations from sentences with zero pronouns from five behavioral experiments conducted in italian by Carminati (2005). |
| Outcome: | The results suggest that human expectations about coreference can be derived from exposure to language, and also indicates features of language models that allow them to better reflect human behavior. |
On the Nature of BERT: Correlating Fine-Tuning and Linguistic Competence (2022.coling-1)
Copied to clipboard
| Challenge: | Several studies on the interpretation of Neural Language Models (NLMs) focus on the linguistic generalization abilities of pre-trained models, but little attention is paid to how the linguistic knowledge of the models changes during fine-tuning. |
| Approach: | They propose to examine whether a wide range of linguistic phenomena are forgotten during fine-tuning and whether it is possible to predict the fine- tuned accuracy solely relying on the assessed linguistic competence. |
| Outcome: | The proposed model can predict the evolution of written language competence of native language learners based on the assessed linguistic competence. |
Unsupervised Attention-based Sentence-Level Meta-Embeddings from Contextualised Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing methods for creating metaembeddings from static word embeddings have been proposed, but they are not tied to a particular downstream task. |
| Approach: | They propose a sentence-level meta-embedding learning method that takes contextualised word embedding models and learns a phrase embeddable that preserves complementary strengths of the input source NLMs. |
| Outcome: | The proposed method outperforms existing methods on semantic textual similarity benchmarks on a supervised baseline and on token-level embeddings. |
Learning Trajectories of Figurative Language for Pre-Trained Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Figures of speech and figures of language are used in everyday communication . however, this imaginative use of words requires a solid understanding of semantics and real-world knowledge. |
| Approach: | They exploit probing tasks to analyse how NLMs recognise figurative language . they find out which layers have a better comprehension of figurativ language based on pre-training data. |
| Outcome: | The proposed model can recognise hyperboles, metaphors, oxymorons and pleonasms . data show which layers have a better comprehension of figurative language . |
From Human Reading to NLM Understanding: Evaluating the Role of Eye-Tracking Data in Encoder-Based Models (2025.acl-long)
Copied to clipboard
| Challenge: | integrating eye-tracking features into Neural Language Models does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space. |
| Approach: | They used eye-gaze data from the Ghent Eye-Tracking Corpus to investigate how integrating knowledge of human reading behavior impacts Neural Language Models. |
| Outcome: | The proposed approach does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space. |